This invention concerns the enhancement of the perceived sharpness of sampled images and video sequences. The invention is particularly suitable for images that have been upconverted from lower resolutions, for images that are defocused and for moving video sequences that are blurred by camera integration.
Images and video sequences are today being captured on devices with a very wide range of resolutions, from a few hundred pixels wide in the case of inexpensive mobile phones, to professional cameras whose resolution in some cases approaches 10,000 pixels wide. Likewise, the resolution of display devices covers a similar range. There is therefore an increasing need to convert between different resolutions, a process that is often referred to as “resizing” in image processing software. Resizing may be classified into two types: downconversion, where the sampling resolution of the output is lower than that of the input, and upconversion, where the sampling resolution of the output is higher than that of the input. Downconversion can usually be accomplished at high quality with a linear filtering operation that ensures that the picture signal meets the Nyquist limit of the new sampling frequency, so that aliasing is avoided. With upconversion, there are two broad approaches. One is to upconvert by repeating input pixels over rectangular blocks of output pictures. This gives a perception of sharpness, but a generally unacceptable overall degradation due to the visibility of the block structure. The other approach is to apply a linear interpolation filter which suppresses frequencies beyond the bandwidth of the input sampling structure. This gives a subjectively smooth and artefact-free output picture, but the problem, especially when comparing with pictures that were created at a higher resolution, is the subjective impression of “softness”. There has for a long time been an interest in enhancing upconverted images to increase their subjective sharpness without introducing the artefacts that arise from sample-repeat upconversion.
Other applications for image enhancement to increase sharpness are defocused pictures, and moving video sequences that are blurred by camera integration. In these cases, the sampling structure can support the portrayal of sharper edges, but the high-frequency signals responsible for subjective sharpness have been attenuated or suppressed. Again, there is interest in enhancing the subjective sharpness of such images.
One method of enhancement has been applied to analogue cathode-ray-tube video displays. For example, S. Yashuda et al in “25-v inch 114-degree Trinitron color picture tube and associated new developments”, IEEE BTR-20, Issue 3, August 1974, pp 193-200, describe a method of beam scan velocity modulation, in which horizontal transitions are sharpened by varying the scan velocity during the transition. Referring to FIG. 1, a video signal (101) is differentiated to obtain a derivative signal (102) which is used to control the beam scan velocity, leading to a variation of the position (103) of the deflected scanning spot from its usual linear ramp characteristic. The effect of this variation on a white window pattern (104) is shown (105) as a narrowing of the pattern. This method underwent several improvements, including the development of versions suitable for digital processing. However, the method is only applicable to horizontal enhancement in beam-scanning displays, which are now largely obsolete.
Another, more generally applicable, method of enhancement involves linear filtering to amplify or “boost” high spatial frequencies. This can be performed as a separate operation or, in the case of upconversion, by modifying the upconversion filter so that the response at higher frequencies is increased. Linear filtering can be effective in the cases of defocusing or motion blur, where high frequencies may be present in the picture but have been attenuated by the defocused lens or by camera integration. The filter can reverse the effect of that attenuation and restore the lost sharpness of the picture. In the case of linear upconversion, there is no benefit in boosting the highest frequencies of the output picture because no signals are present at those frequencies. However, subjective sharpness can sometimes be increased by boosting the highest input frequencies either before or after upconversion.
Linear filtering has various disadvantages. One disadvantage is that noise on the input picture can be amplified. This is because noise generally makes a greater relative contribution to the higher frequency signals, and can therefore be disproportionately amplified. Another disadvantage is that high-pass filtering can increase the dynamic range of the signal, leading to the possibility of clipping and, ironically, consequential loss of detail. An example of this is given in FIG. 2, in which a linear filter is applied to a rising and a falling edge in the input video (201) which has the effect of steepening the transition but which also leads to overshoots in the output signal (202).